Improving Evaluation and System Design through the Use of Off-nominal Testing: a Methodology for Scenario Development

A major challenge facing system designers is to ensure that a system that has been successfully evaluated in the laboratory will be successfully fielded. Off-nominal testing, in which unexpected conditions are evaluated, is proposed as a method to meet this challenge. Off-nominal testing allows for increased understanding of the humanmachine system under evaluation, may uncover design issues that can be addressed, and may help determine training issues and procedures. An experimental methodology for conducting off-nominal testing in human-in-theloop evaluations is presented. The methodology involves developing issues to be tested and off-nominal events addressing those issues, and estimating their disruptiveness on the test participant and the other test trials. Low and moderately disruptive off-nominal events are then incorporated into an experimental design while still allowing for the accurate estimation of dependent measures in the nominal trials. A single final trial can be incorporated for highly disruptive or truly surprising off-nominal events.